The dramatic increase in the use of wireless network technologies requires new approaches both to network design, and to the applications which depend on them. The functionality, performance, and scope of mobile applications are no less complex than tethered, wired network applications. In addition, they require many more input combinations to adequately test the interaction of mobility-related application features with mobile device features essential to wireless applications. When a user of a cell phone or personal digital assistant (PDA) taps a few keys to get current stock quotes or a weather report, a very large and complex collection of application software, operating system software, network equipment, databases, telecommunications equipment, and radio transmission equipment must cooperate to service that request. End-to-end testing means exercising the complete round trip from the end-user interface through a wireless and wired network to the application server systems and back to the end-user. The response time and results the end-user sees are affected not only by all these piece parts, but also by how many other end-users are active and what they are doing, as well as that end-user's location and rate of movement. Signal strength varies both with speed of movement and geographic position, airlinks can be dropped or handed-off to equipment with different protocols, and, for location-based services, the end-user's actual, real-time, physical location is a critical input.
Designing wireless packet-based mobile networks requires the designer to simultaneously consider several aspects of network structure and behavior. The designer must consider burst characteristics of packet traffic, bandwidth dimensioning for real time applications like voice, data, steaming video, and Internet applications. Present-day routing algorithms use not only the shortest path but also the less congested, priority, resource reservation mechanisms, and so on. Thus, delivering reliable mobile applications is not simply a matter of providing sufficient network bandwidth. Link topologies, traffic characterization, RF propagation/dimensioning models, economic and tariff models must also be considered. Before deployment of applications dependent on such sophisticated network technologies, the networks and their features must be accurately and efficiently tested to determine, among other things, network performance and/or reliability in the context of application-specific loads.
Moreover, as software and hardware designs for wireless communication devices have become increasingly complicated, it is often difficult, risky, costly and/or inefficient to verify the performance, scalability, and reliability of a wireless packet-based mobile network carrying data services for a multitude of devices. Typically, a communication device includes thousands of hardware components and millions of lines of software. Such complexity makes it difficult to predict how these components will influence one another during operation. For example, the type of protocol the device will use to communicate with over a network may have significant impact on the hardware and/or software design. Furthermore, the interaction of several communication devices over a wireless network may result in field performance far different from that seen in or estimated for a pairwise point-to-point scenario
EXHIBIT 1Scope of Testing Needed to Reveal Typical Mobile Application Failure ModesMobile Application FailureMUDBaseMSC/TCP/IPAppMode by Component21StationPSTNNetworkServersMUD Application ClientUnder Test 16FunctionalityXResponse TimeXMUD 21 Resource UtilizationXAirlink InterfaceQOS Edge CombinationsXXIn-cluster Hand OffsXXXMultiple Base St ProtocolXXXRoamingXXXLocation ServicesOn device GPSXXTriangulationXXServer InteractionIncorrect outXXAbendXXServer ExceptionXXConfigurationXXXXXBase StationOperations/Administration/MaintenXXXBackground load (“breathing”)XXXPacket Load/Edge CombinationsXXXWeather, solar, etc.XXXApplication Server 6FunctionalityXResponse TimeXServer Resource UtilizationXBilling/Provisioning/SecurityXBackground processor contentionXDispatch/AllocationXXBackground IP LoadXXClient transaction saturationXXXXXEnd to endResponse timeXXXXXCapacityXXXXXReliabilityXXXXXAvailabilityXXXXXGeographic CoverageXXXXX
Exhibit 1 shows the relationship between the kinds of failures that can occur in a mobile application network and the scope of testing necessary to produce those failures (development testing seeks to cause failures so that they may be corrected before general release.) The Xs show which components must be tested together to reveal a failure mode. For example, suppose a bug in a Mobile User Device (MUD) application software 16 is sensitive to airlink QOS (Quality of Service) levels only seen when the MUD 21 is moving at 60 miles per hour and the cell cluster range is contracting (“breathing”) due to an increasing load. The test environment must control at least the MUD 21 and the Base Station to evaluate the application's response to this situation and to reveal what failure-causing defects, if any, are present.
Although the operational field installation of a complex network system would seem to be an ideal test bed for the applications it supports, using an operational network for testing is almost never practical or prudent. This would be like using a metropolitan expressway for automotive crash testing at rush hour. Even if an expressway could be cleared, it would be an ineffective test environment because of obstacles to controllability and repeatability. For example, the ideal test environment for an automotive telematics system is a million cars and drivers with active transceivers using actual satellite links, accessing actual database servers, and transmitting results back to each vehicle (i.e., the actual field system.) Supporting this idealized development testing would further require each of the million drivers to drive a preplanned route, entering preplanned tests at predetermined moments, over days, weeks, and months. However, such an approach is never practical. Controlling the test setup, varying test parameters, or rerunning tests on a new or modified system would be cumbersome and prohibitively costly, if it could be accomplished at all. Current best practice is to test components in isolation, followed by limited inter-component testing in a scaled-down lab. Some automated load testing using low-fidelity software emulation may be done. Finally, the testers conduct a few end-to-end functional tests, often manually, and often in only a few field locations. Even this limited testing is very costly and often does not achieve the desired reliability.